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In Ref. ! l|, we invoked the powerful results of Shore and 
Johnson (SJ) Q, who showed that, under quite general 
circumstances, the least-biased way to infer a probability 
distribution {p,} is to maximize the Boltzmann-Gibbs 
relative entropy 

H = - Y^Pi^gipi/qi) ( 1 ) 
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(or any function that is monotonic with this entropy), 
under constraints where is the prior distribution that 
contains any foreknowledge of the system. In Ref. |lj, 
we showed that mathematical forms of H that are not 
monotonic with Eq. o which we call noncanonical 
entropies - lead to unwarranted biases. In his comment 
to our Ref. [U], Tsallis objects to this argument on three 
grounds. We show here the flaws in his objections. 

First, Tsallis contends that “nonadditive entropies 
emerge from strong correlations which are out of the SJ 
hypothesis” adding that the “SJ axiomatic framework 
addresses essentially systems for which it is possible to 
provide information about their subsystems without pro¬ 
viding information about the interactions between the 
subsystems, which clearly is not the case where nonad¬ 
ditive entropies are to be used”. These statements are 
incorrect. SJ is not limited in any way to small interac¬ 
tions or weak correlations; it can handle interactions of 
any strength. 

Standard methods of statistical physics - which are 
grounded in Eq. 0 - provide a clear recipe for treating 
correlated systems while assuming no initial subsystem 
correlations. For example, we don’t need a special form 
of Eq. 0 in order to build spin-spin correlations into 
an Ising model. On the contrary, the Boltzmann weights 
for a spin-spin-correlated Ising model are constructed by 
assuming that spin correlations originate from the data 
(which is used to constrain Eq. 0 ). In the absence 
of spin-spin correlations, standard statistical physics re¬ 
turns an Ising model with decoupled spins, as it should. 
On the other hand, if we choose to assume a priori some 
particular coupling between the spins that do not orig¬ 
inate from data, then SJ prove that it should be intro¬ 
duced through {q{\ exactly as it appears in Eq. 0 . In 


particular, models of power laws can only arise in a prin¬ 
cipled way either from data constraints or from {<p} in 
conjunction with Eq. 0. There is no principled basis 
for power laws that can be obtained bv re-assigning the 
meaning of H and changing its form [llj. 

Second, Tsallis asserts that “the SJ axioms, demanding 
as they do, system independence, ...”. This is not correct. 
The SJ inference procedure is not limited to independent 
systems. Rather, SJ asserts that when systems are in¬ 
dependent of each other, then the joint outcome for two 
independent systems must be the product of marginal 
probabilities if data are provided for systems indepen¬ 
dently [jj. SJ is otherwise perfectly applicable broadly 
across situations not involving independent systems. 

Third, Tsallis asserts that “This is the deep reason 
why, in the presence of strong correlations, the BG en¬ 
tropy is generically not physically appropriate anymore 
since it violates the thermodynamical requirement of en- 
tropic extensivity.” In this statement, Tsallis is incor¬ 
rectly asserting that extensivity must be a foundational 
property from which the functional form of the entropy 
follows. In fact, the logic is just the opposite. Extensivity 
- or not - of an entropy is the outcome of an inference 
problem at hand, not its input. Certainly throughout 
much of equilibrium thermodynamics, extensivity hap¬ 
pens to hold. But that is a matter of the particulars of 
that particular class of applications. 

Said differently, this argument confuses the distinc¬ 
tion between pre-maximization and post-maximization 
entropies. SJ focus on pre-maximization. SJ seek a 
functional H that, upon maximization, achieves certain 
properties required for drawing consistent unbiased in¬ 
ferences. At this stage, no system property (such as 
how entropy scales with system size) is yet relevant. 
This is just establishing a very general inference prin¬ 
ciple. However, once maximization has been performed, 
H({pi = p*}) = — JAp* lnp* is an entropy function that 
may be used to make predictions about physical systems 
including how properties scale with system size. The SJ 
argument is agnostic about whether extensivity holds or 
not for the post-maximization entropy H({pi = p*}). 
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In short, the great power of the SJ arguments is in 
showing that Eq. 0 is an extremely broad and deep 
result, applicable across all matters of inference of prob¬ 
ability distributions, given only priors (g^) and given new 
information. The power of the SJ arguments is that they 
apply upstream of any particular application, whether it 
should involve equilibria or dynamics, materials or in¬ 
formational channel capacities or other, weak or strong 
correlations, extensivity or not, or any other particular¬ 
ity. We are assured by Ref. [2| that no other form of 
entropy function - beyond those monotonic with H 
can generate unbiased inferences. 
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